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(57) Claims 

1 Method for controlling a manipulator having a function of providing sensory 
information, characterized in that a sensory information sensor is disposed in a work 
environment in which is situated a work manipulator that moves according to instruction 
from an operator and designed to provide, on the basis of sensory information sensed by 
said sensory information sensor, information to a sensory organ of an operator situated in 
a control room outside said work environment, by means of a sensory providing device 
that tracks the movements of said operator; employing an operator sensing device that 
measures in real time the position and attitude of said operator, and an actuating device 
that on the basis of the output of said operator sensing device controls the position and 
attitude of said sensory information sensor, in order to cause position and attitude relating 
to three-dimensional coordinates at an arbitrary fixed location of said sensory information 
sensor to follow a position and attitude relating to three-dimensional coordinates on said 
control room side, [said coordinates] corresponding to three-dimensional coordinates of 
said operator; said sensory information sensor is at least a visual information sensor for 
sensing visual information in said work environment, an auditory information sensor for 
sensing auditory information in said work environment, and a force sensor for sensing 
force acting upon said work manipulator; said sensory providing device comprises at least 
as visual providing device, an auditory providing device, and a skin stimulating device; 
said visual providing device providing the operator with visual information based on 
visual information detected by said visual sensing device with visual information around 
said operator obstructed; said auditory providing device providing the operator with 
auditory information based on auditory information detected by said auditory sensing 
device with auditory information around said operator obstructed; and information based 
on information sensed by said force sensor being provided to the operator through change 
in skin stimulation by said skin stimulating device. 
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Detailed Description of the Invention 

This invention relates to a method for controlling a manipulator. 

Hazardous operations such as those in a nuclear reactor or unsanitary operations 
are commonly carried out using a remote operation type manipulator. With this kind of 
manipulator, typically, the work environment is separated from a control room by means 
of a barrier chamber or barrier window; a work manipulator is installed in the work 
environment, and a control manipulator is installed in the control room, with the control 
manipulator being operated by an operator to cause the work manipulator to perform 
tasks by following the movements of the control manipulator. In such instances, the 
object being manipulated and the arm of the work manipulator in a work site such as a 
nuclear reactor are sensed by means of a visual sensor such as a television camera or the 
like, and this [image] is presented to the operator displayed on a television in the control 
room; or tactile or pressure sense when an object is grasped by the work manipulator, or 
locational sense as to the location of the arm of the work manipulator, is fed back to the 
control manipulator on the control room side, or an exoskeleton-like case is placed 
around the arm of the operator, and feedback provided thereto. 

With such control methods, movement of the arm of the displacing work 
manipulator is observed from a fixed location by means of a television camera or the like, 
and for reasons such as failure to organically associate information to be communicated 
to the operator through his or her senses with information input from his or her arm, or 
where information is provided visually, the need to constantly compare the status of the 
work manipulator on the work side from a television screen with the status of the control 
manipulator on the control room side, or for some other reason, there is a marked drop in 
work efficiency and increase in operator fatigue as compared to direct manipulation of the 
work manipulator on the work side. 

With the foregoing in view, it is an object of this invention to provide a method 
for controlling a manipulator, whereby the operator may be provided with a sense of 
realism, the operator per se may be made to experience the impression of being actually 
present within the work environment, and the operator may improve his or her work 
efficiency to a level on part with that of directly manipulating an object. 
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To achieve this object, the method for controlling a manipulator having a function 
of providing sensory information according to the present invention is characterized in 
that a sensory information sensor is disposed in a work environment in which is situated a 
work manipulator that moves according to instruction from an operator and designed to 
provide, on the basis of sensory information sensed by said sensory information sensor, 
information to a sensory organ of an operator situated in a control room outside said work 
environment, by means of a sensory providing device that tracks the movements of said 
operator; employing an operator sensing device that measures in real time the position 
and attitude of said operator, and an actuating device that on the basis of the output of 
said operator sensing device controls the position and attitude of said sensory information 
sensor, in order to cause position and attitude relating to three-dimensional coordinates at 
an arbitrary fixed location of said sensory information sensor to follow a position and 
attitude relating to three-dimensional coordinates on said control room side, [said 
coordinates] corresponding to three-dimensional coordinates of said operator; said 
sensory information sensor is at least a visual information sensor for sensing visual 
information in said work environment, an auditory information sensor for sensing 
auditory information in said work environment, and a force sensor for sensing force 
acting upon said work manipulator; said sensory providing device comprises at least as 
visual providing device, an auditory providing device, and a skin stimulating device; said 
visual providing device providing the operator with visual information based on visual 
information detected by said visual sensing device with visual information around said 
operator obstructed; said auditory providing device providing the operator with auditory 
information based on auditory information detected by said auditory sensing device with 
auditory information around said operator obstructed; and information based on 
information sensed by said force sensor being provided to the operator through change in 
skin stimulation by said skin stimulating device. 

A fuller understanding of the invention is provided through the following 
description of the embodiment with reference to the accompanying drawings. 

Fig. 1 illustrates the basic principle of this invention. According to the 
manipulator control method herein, a dummy 2 is disposed in a work environment 1 in 
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which an actual task A will be carried out, and an operator 4 is situated in a control room 
3 outside the work environment 1 . Dummy 2 comprises a work manipulator 5 and a 
sensory information sensor 6. Work manipulator 5 is an effector of dummy 2 for 
performing an actual task 7. Sensory information sensor 6 senses sensory information 
such as visual conditions within work environment 1, sound within work environment 1, 
contact force against an object being manipulated, tactile sensation, or the like. The 
operator 4 in control room 3 is not provided with screened environmental information, 
but rather the sense organs 10, such as the eyes, ears or skin of operator 4, are presented 
exclusively with information about the work environment 1 sensed by the sensory 
information sensor 6 and processed by a processing device 30. Movement of the dummy 
2, i.e., motion of the work manipulator 5 and/or the sensory information sensor 6, 
specifically position and attitude relating to three-dimensional coordinates in an arbitrary 
fixed location, are made to conform to movements of the body and limbs of the operator 
4, i.e. to position and attitude relating to three-dimensional coordinates on the control 
room side corresponding to the aforementioned three-dimensional coordinates. By so 
doing, the operator is given the impression of being actually present within the work 
environment, thereby affording effects similar to direct manipulation by the operator. 

Fig. 2 shows a specific arrangement embodying the general principle of the 
invention described hereinabove. Specifically, a dummy 2 is disposed in a work 
environment 1, while an operator 4 is situated in a control room 3 outside the work 
environment 1. Dummy 2 is supported on a fixed member 8 via an actuating device 7 so 
as to be able to undergo displacement in three orthogonal directions x, y, z by means of 
operation of the actuating device 7, and also to change direction through rotation of 
rotation angles indicated by <t> and 0. 

Dummy 2 is equipped with sensory information sensors 6 composed of television 
cameras 9a, 9b and microphones 1 la, 1 lb, situated at locations corresponding to the 
human eyes and ears, and with a work manipulator 5. Work manipulator 5 is equipped 
with force sensors 12a, 12b such as strain gauges or the like. 

The control room 3, meanwhile, is equipped with a sensory providing device 13, 
operator sensing device 14, and operator arm sensing device 15. 
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Sensory providing device 13 comprises a three-dimensional television display 16, 
a headphone or in-ear speaker 17, and a skin stimulating device 18. 

As shown in Fig. 2 and Fig. 3, three-dimensional television display 16 is 
composed of a left eye display 19 and right eye display 20, each consisting of a television, 
disposed within a case 22. The three-dimensional television display 16 is used by being 
placed covering the eyes of the operator 4, with the left eye display 19 juxtaposed to the 
left eye of the operator 4 and the right eye display 20 juxtaposed to the right eye of the 
operator 4, whereby image information entering via the two eyes of the operator is 
perceived as a three-dimensional image by virtue of being synthesized in the nerve center 
40 of the operator. Images projected onto the left eye display 19 are produced by video 
conversion by processing device 30 of images taken by television camera 9a, and images 
projected onto the right eye display 20 are produced by video conversion by processing 
device 30 of images taken by television camera 9b. Case 22 covers the area around the 
eyes of the operator 4, blocking out information from the control room environment. 

Headphone 17 blocks out information from the environment while providing the 
auditory organs (ears) of the operator with information relating to audible noise in the 
work environment 1; as shown in Fig. 2 and Fig. 4, noise put out from the left ear speaker 
24a is produced by audio conversion by processing device 30 of auditory information in 
the work environment 1 sensed by microphone 11a, and noise put out from the right ear 
speaker 24b is produced by audio conversion by processing device 30 of auditory 
information in the work environment 1 sensed by microphone 1 lb. Skin stimulating 
device 18 is attached in proximity to the skin of the arm, etc. of the operator 4, and 
comprises a electrical skin stimulating device 25 and an oscillating skin stimulating 
device 26. In response to an input, electrical skin stimulating device 25 outputs electrical 
current which passes to the skin of the operator, thereby stimulating the skin and 
producing a sensation of exerting force on the work manipulator 5 or the like; oscillating 
skin stimulating device 26 outputs oscillations in response to an input, [which 
oscillations] are provided to the skin of the operator, producing a sensation of exerting 
force on the work manipulator 5 or the like. Each [of these devices] has as input 
information about the inside of the work environment 1 sensed by the strain gauges or 
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other force sensors 12a, 12b attached to the work manipulator 5, and processed by 
processing device 30. 

The operator sensing device 14 comprises television cameras 27a, 27b and/or cell 
spots, and senses the position and orientation of the operator 4; this information is sent to 
actuating device 7 via processing device 30. Presented with this input signal, actuating 
device 7 decides upon a position and orientation for dummy 2, i.e. television cameras 9a, 
9b and microphones 1 la, 1 lb, which conform [to those of] the operator. 

The operator arm sensing device 15 comprises a goniometer 28; arm movements 
of the operator 4 are measured by the goniometer 28, and this information is sent to the 
manipulator actuating device 29 via the processing device 30, whereupon on the basis of 
this input signal, manipulator actuating device 29 decides upon a position and orientation 
for work manipulator 5 which conform [to those of] the control manipulator. 

With this arrangement, movements of operator 4 are measured, in real time and 
without restricting movement of the operator 4, by means of television cameras 27a, 27b, 
cell spots or other such operator sensing device 14, and a goniometer 28 Or other such 
operator arm sensing device 15, the dummy 2 being caused to move in coordination with 
movements of the operator measured in this manner, and images of changes in the 
external world (images and sounds) accompanying movement of the dummy 2, tactile 
sensations, and other conditions of the external world being fed back to the operator 4 by 
means of three-dimensional television 16, headphone 17, and skin stimulating device 18. 
The operator 4 is isolated from environmental information within the control room 3, and 
can receive only sensory information within the work environment 1, which sensory 
information changes in response to his or her own movements. In this way, the operator 
may be given the impression of being actually present within the work environment, 
improving work efficiency to a level comparable with direct manipulation by the 
operator. 

In particular, force exerted on the slave manipulator is not transmitted as 
unmodified force to the master manipulator; rather, force exerted on the slave 
manipulator is transmitted to the operator as a skin stimulus, thereby avoiding danger to 
and fatigue of the operator. That is, where force exerted on the slave manipulator is 
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transmitted as unmodified force to the master manipulator, if the master manipulator 
should be subjected to a strong force due to an accident or the like, this strong force will 
be transmitted to the operator through the master manipulator, which can be extremely 
dangerous for the operator. 

Depending on the nature of the work, the slave manipulator may need to be 
subjected to strong force, and in such instances the operator must put out [force to 
counter?] this strong force, resulting in marked operator fatigue. 

Danger to and fatigue of the operator can be avoided by the present invention. 



Brief Description of the Drawings 

Fig. 1 is a block diagram illustrating the general principle of this invention; Fig.2 
is an illustration of a manipulator control device; Fig. 3 is an illustration of a three- 
dimensional display; and Fig. 4 is an illustration of a headphone. 

1 ... work environment, 2 ... dummy, 4 ... operator, 5 ... work manipulator, 6 ... 
sensory information sensor, 9a, 9b ... television cameras, 11a, 1 lb ... microphones, 
16 ... three-dimensional television display, 17 ... headphone, 30 ... processing device 



J 

2 



10 



A 



Fig. 1 

40 
) 



•30 



K 



20 



I 



~3 



7 



9 




10 



Fig. 4 
I to 




4 24b 



11 



